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OPTICAL IN SITU INFORMATION SYSTEM 

Background of the Invention 

1 . Field of the Invention 

The present invention relates to an optical in situ 
5 information system. In particular, the present invention 
relates to a system that presents a wide variety of 
information (synthetic objects) in situ with a field of 
view of a view looking through an optical system, such as 
a microscope or a different type of viewing device with 
10 arbitrary magnification. 

2 . Description of the Related Art 

There are several types of conventional systems that 
enable a viewer to view synthetically produced pictures. 
One such type of conventional system is a virtual reality 

15 system. In such systems, stereo pairs of a scene are 
totally synthesized (generated) with computers, and then 
viewed through a viewer with a binocular viewing device. 
In virtual reality systems, no attempt is made to overlay 
the synthetic view onto a view of real, bright objects. 

20 In fact, typically virtual reality systems are used in a 
dark environment, so that synthetic images of high 
brightness are not needed. 

Another type of conventional system is a head-mounted 
display, as used by pilots and the like. In such 

25 systems, the objects in the field of view of the pilot 
are so distant that they do not produce a perceptible 
stereo-effect. Conseguently , the field of view is 
essentially a two-dimensional plane without any depth 
perception. As a result, the synthetic pictures 

30 superimposed over the pilot's view do not have any 
reguirements of depth perception. 
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Still another type of conventional system is a 
display built into surgical microscopes. Such systems 
are described, for example, in U.S. Patent No. 4,202,037, 
issued to E . M . Glaser, and in U.S. Patent No. 4,722,056, 
5 issued to Roberts et al. Glaser describes a system for 
simultaneous viewing (without aligned overlay) of 
computer generated information in the field of view of a 
surgical microscope. Roberts describes a method of 
producing aligned overlays. This method is limited to 
10 overlays of two-dimensional pictures correctly aligned 
only in the focal plane of the microscope. In principle, 
Roberts 1 method is similar to the head mounted displays 
used by pilots, and only provides a monoscopic view. 

Summary of the Invention 

15 One object of the present invention is to provide a 

sufficient light capability to view a synthetic image 
against a bright real image. 

Another object of the present invention is to provide 
a stereoscopic view of a combined image of a synthetic 
20 image with a real image. 

Still another object of the present invention is to 
provide a stereoscopic, combined image of a three- 
dimensional synthetic image with a three-dimensional real 
image . 

25 The present invention relates to a method by which 

computer generated images of synthetic three dimensional 
objects can be correctly overlayed and aligned over the 
images of real, three-dimensional objects in the field of 
view of a viewer. The embodiments described herein also 

3 0 include many novel features and techniques which will 
improve brightness, contrast, resolution, ease of 
adjustment, and operation over the conventional systems 
for aligned and unaligned overlays of two dimensional as 
well as three dimensional objects. 

35 The method includes a step of generating a synthetic 

stereoscopic pair of pictures of a three-dimensional 
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synthetic object which correspond to the synthetically 
produced three-dimensional picture. The method also 
includes a step of respectively combining the synthetic 
stereoscopic pair of pictures with a real stereoscopic 
5 pair of pictures which correspond to respective images 
the real object, the real stereoscopic pair of pictures 
being respectively obtained from two oculars of a 
microscope. By this method, the synthetically produced 
picture is made to overlay the images of the real object 
10 over a three-dimensional space and not just over a single 
focal plane. 

Brief Description of the Drawings 

These and other objects and advantages of the 
invention will become more fully apparent from the 
15 following detailed description when read in conjunction 
with the accompanying drawings with like reference 
numerals indicating corresponding parts throughout, and 
wherein: 

Figure 1 shows a conventional viewing system using 
20 a microscope; 

Figure 2 shows a conventional stereoscopic viewing 
system using a microscope in order to achieve a 
stereoscopic image ; 

Figure 3 shows a system for generating an overlay of 
25 images in a single focal plane according to a first 
embodiment of the invention; 

Figure 4 shows a system for generating a stereoscopic 
image according to either a second or third embodiment of 
the invention; and 
30 Figures 5A and 5B shows a difference in the mapping 

of LCD pixels for a synthetic image between the second 
and third embodiments, with respect to a reference plane 
and a different plane. 
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Detailed Des cription of the Preferred Embodiments 

The present invention relates to an optical in situ 
information system, which presents a wide variety of 
information (synthetic objects) in situ with the field of 
5 view of a viewer looking through an optical system such 
as a microscope or other type of viewing device with 
arbitrary magnification. The term "microscope" used 
throughout will be interpreted to include microscopes, 
binoculars, as well as optical systems attached to 

10 goggles. The term "synthetics objects" is used 

throughout to emphasize that the information displayed by 
the present invention is synthesized with computers and 
electronics, and it may represent two-dimensional as well 
as three-dimensional objects. 

15 In a usual operation of a microscope, light from an 

object (i.e., source of information) in the object space 
of the microscope enters the microscope objective and 
exits the oculars of the microscope for viewing the image 
of the object. In an extension of this concept, light is 

20 directed towards an object 100 by a light source 105. A 
portion of the light directed towards the object 100 
reflects off of the object 110 and enters a microscope 
110. 

The light that enters the microscope 110 is split by 

2 5 a beam splitter 12 0, after passing through oculars 13 0. 

One portion of the output from the beam splitter 12 0 is 
directed to an imaging device 140, such as an electronic 
camera, as shown in Figure 1. The other portion of light 
output from the beam splitter 120 is directed to an eye 
30 160 of a viewer. The electronic camera 140 may be used 
for either viewing the image on a TV monitor or for 
electronic analysis of the image. 

For stereoscopic viewing of the object, the object 
is imaged from two different angles, and viewed through 

3 5 two oculars to form a stereoscopic image of the object. 

Figure 2 shows a conventional use of a first microscope 
210 and a second microscope 22 0, which together provide 
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stereo signals to a TV monitor 230, in order to obtain a 
stereoscopic image of the object. 

A first embodiment of the invention corresponds to 
a system for simultaneous viewing of a real object and a 
5 synthetic object. In the first embodiment, light from 
two separate sources is combined by an optical combiner 
310, as shown in Figure 3. The first source of light is 
a synthetic image, created by a computer 34 0 that outputs 
a video signal to an LCD 3 50, where the LCD 3 50 is 

10 illuminated by a light source 360. The output of the 
LCD 3 50 is combined with light received from the object 
100 via the objective lenses 3 20. The combined light 
output from the optical combiner 310 passes through an 
ocular 330, which focuses the light so that both objects 

15 can be viewed simultaneously in the field of view of a 
viewer 160. In Figure 3, the ocular 3 30 and the 
objective 32 0 are both shown as having two separate lens 
spaced apart from each other, but one of ordinary skill 
in the art will recognize that many different types of 

20 oculars and objectives may be utilized while keeping 
within the scope of the invention. 

In the first embodiment, one of the images being 
viewed corresponds to an image from a real object, while 
the other image being viewed corresponds to a 

25 synthetically produced picture or text input through 
electronics, such as created using a liquid crystal 
display (LCD) . In the first embodiment as described 
above, simultaneous viewing of a real object and a 
synthetic object is thereby achieved. By the use of the 

30 LCD 350 with the appropriate, amount of light applied to 
it by the light source 360,. one is able to clearly view 
a synthetic image overlayed with a bright, real image 
from the object 100. 

In order to achieve stereoscopic viewing, a second 

35 embodiment of the invention uses two separate systems - 
one for the left eye and one for the right eye, with the 
two images paraf ocalized to produce an overlay of images 
onto a single focal plane for viewing by a viewer. 
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Figure 4 shows a system according to a second embodiment 
of the invention, in which stereoscopic viewing is 
achieved/ 

The second embodiment includes a first computer 410A, 
5 which outputs electrical signals (for example, a video 
signals) responding to a "left eye synthetic image" to 
LCD and Optics Unit 420A. LCD and Optics Unit 420A 
includes a matrix array of LCD pixels, as well as optics 
for outputting the light from the pixel array at a 

10 particular magnification and brightness. LCD and Optics 
Unit 4 2 OA receives light from Light Source 4 30A f via a 
fiber optic cable (or cables) or the like, and outputs an 
illuminated synthetic image to beam combiner 4 4 OA. 

Illuminators 4 50A and 450B provide light in a 

15 direction of a real object 490. Illuminators may be 
configured as lamps that output light across a wide 
optical spectrum. Some of the light incident on the real 
object 490 is reflected in a direction towards Objectives 
460A and 460B. Objective 460A is an objective lens for 

20 focusing light to be eventually directed to a left eye of 
a viewer, and objective 460B is an objective lens for 
focusing light to be eventually directed to a right eye 
of a viewer. 

Light output from objective 460A is directed to beam 
25 combiner 440A, where the "real image" from the real 
object is combined with the "synthetic image" received 
from the LCD and Optics Unit 420A. The combined 
"synthetic and real" light passes through ocular lenses 
470A, and is focused onto a left eye 495A of the viewer. 
3 0 Similarly, reflected light is also received by 

objective 460B, and is directed to beam combiner 440B. 
Second computer 410B creates a video signal for a "right 
eye synthetic image", which is sent to LCD and Optics 
Unit 42 0B. LCD and Optics Unit 420B also receives light 
35 from Light Source 430B, via a fiber optic connection or 
the like. The illuminated synthetic image output from 
LCD and optics Unit 4 2 0B is sent to beam combiner 4 4 0B, 
where it is combined with the real image corresponding to 
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light received from the real object. In the second 
embodiment, the LCDs of LCD and Optics Units 420A and 
4 2 OB correspond to a matrix of pixels, arranged in a two- 
dimensional grid. The combined light from beam combiner 
5 440B passes through ocular lenses 470B, and is focused 
onto a right eye 495B of the viewer. 

By the system according to the second embodiment, the 
viewer can see a stereoscopic viewing of a real image 
superimposed with a synthetic image. Such a 

10 configuration is useful in many medical procedures. For 
example, a CATS CAN or a selected portion thereof, as a 
synthetic image, can be superimposed onto a real image 
corresponding to an actual patient situated in a viewing 
region of the system according to the second embodiment. 
15 In the second embodiment, a calibration is performed 

so that the synthetic image can be properly imaged onto 
one or more pixels of the LCD array in the LCD and Optics 
Units. That calibration may be performed in a number of 
ways, such as providing a test image at a particular 
20 distance and angle with respect to the viewing system, 
and then determining where that test image appears on an 
X-Y array of LCDs. Based on this information, which is 
taken at a variety of angles of objects with respect to 
the viewing system, a synthetic image can be properly 
25 superimposed onto a real image by knowledge of the actual 
position of the real object with respect to the viewing 
system as well as knowledge of which pixel (or pixels) of 
the LCD array would be illuminated by the object at that 
position. 

50 Xn the second embodiment, - a reference plane 

corresponds to the plane of the LCD array of pixels, and 
the synthetic image is determined based only on where it 
would appear on the reference plane, and no other plane. 
Thus, the combined image of the real image and the 

)5 synthetic image is exactly valid for a real object on the 
reference plane, and may not provide an acceptable 
overlay of the synthetic image onto the real image when 
the real object is not on the reference plane. The 



WO 98/13716 PCT/US97/17264 

-8- 

second embodiment does not require complex calculations, 
however, and may be acceptable for some situations where 
an exact overlay is not necessary. However, where a 
precise overly is required, such as in complex surgery, 
5 for example, a different scheme is desired. 

In that regard, a third embodiment of the invention 
produces a three-dimensional overlay of a synthetically 
produced picture over a three-dimensional image of the 
real object. The third embodiment has a structure 
10 similar to the second embodiment of Figure 4, but the 
computer in the third embodiment provides the means for 
creating a three-dimensional synthetic image that can be 
superimposed onto the three dimensional real image, and 
which will appear realistic when the real object is 
15 viewed from any angle and/ or distance. 

First, a synthetic stereoscopic (left and right eye) 
pair of pictures of the synthetic object is calculated by 
the computers 410A and 410B, and then displayed on the 
LCDs 4 2 OA and 4 2 OB, taking into account the parameters of 
2 0 the viewing optical system. These parameters may be 
calculated from pre-calibrated correspondence between 
points in the three-dimensional object space and points 
in a plane of the LCDs. This method is illustrated by 
Figures 5A and 5B, which show a correspondence between 
25 the real object space and the LCD space. Referring now 
to Figure 5B, as the real object moves from a reference 
plane 510 to a different plane 520 (i.e., at a different 
distance with respect to the microscope), a point A 1 on 
the different plane 520 maps to a different pixel 
30 position B on the reference plane 510. 

In the second embodiment, the synthetic object is 
considered to be a two dimensional object in the focal 
plane of the microscope. Therefore, in the calibration 
process of the second embodiment, a correspondence 
35 between the points in the focal plane of the microscope 
and points in the LCD plane is determined. In case the 
actual synthetic object is a three dimensional object, 
either a cross-section of this three dimensional object 
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or a collapsed projection of this three dimensional 
object is displayed in the LCDs to produce an overlay in 
the focal plane of the microscope. 

In the second embodiment , the point A 1 on the 
5 different plane 520 would be collapsed to the point A on 
the reference plane 510, since the synthetic image is 
only overlayed in a two-dimensional manner onto the real 
image. Figure 5A shows the locations of a pixel A and an 
adjacent pixel B on an LCD plane, where the LCD plane 

10 includes a plurality of LCDs (not shown) positioned in a 
matrix arrangement. The intersection of the dashed and 
dotted lines in Figure 5B corresponds to a location of 
the viewing system with respect to the reference plane 
510 and the different plane 520. 

15 if the real object is on the reference plane, then 

the pixel illuminated by the system according to the 
second embodiment would correspond to the pixel 
illuminated by the third embodiment. However, if the 
object is moved a particular distance away from the 

20 reference plane (i.e., either closer to or farther away 
from the microscope) , then the pixel illuminated by the 
second embodiment would be different from the pixel 
illuminated by the third embodiment, as shown in Figure 
5B. In essence, the LCD array is configured to make the 

25 synthetic image appear at a particular distance with 
respect to the microscope, with that particular distance 
corresponding to the reference plane. 

In the third embodiment, the means for determining 
the mapping of a three-dimensional real space onto a two- 

30 dimensional LCD pixel array is preferably performed 
beforehand by calibration , based on real images received 
from a real test object, and by taking into account the 
distance and angle of the viewing system with respect to 
the real test object. At the same time, the computers 

35 410A and 41 0B determine, for the X-Y grid of LCD pixels 
on the LCDs 4 2 OA and 4 2 0B, where the real object would 
appear (i.e., which pixel in the LCD array should be 
illuminated) in order to create an appropriate three- 
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dimensional synthetic image to be overlayed onto the 
three-dimensional real image from the object 490. 

The system according to the third embodiment takes 
into account the depth and angle of the viewing system, 
5 while the system according the second embodiment 
collapses the synthetic image onto a single (reference) 
plane. m other words, in the third embodiment, the 
X,Y,Z object space corresponds to a particular pixel 
number (X,., Y L coordinate) on the left LCD array, and a 
10 particular pixel number (X R , Y R ) on the right LCD array. 
The third embodiment provides a three dimensional mapping 
onto the useful space, while the second embodiment 
provides a two dimensional mapping of the synthetic image 
onto the useful space. in the third embodiment, the 
15 position of the LCD with respect to the optics in the LCD 
and optics Units 420A and 420B can be adjusted to make 
light from the LCD appear (that is used to create the 
synthetic image) to come from a suitable plane in the 
real object (such as in the middle of the real object, or 
20 at another plane, which in the third embodiment is 
operator-adjustable) . The system thus provides a more 
comfortable viewing of the overlay of the synthetic image 
with the real image. 

In the second and third embodiments, the oculars may 
also be adjusted so that light appears to be at a 
particular distance away, and not necessarily at an 
"infinite" distance away. That way, a realistic three- 
dimensional viewing of an object together with a three- 
dimensional synthetic image may be achieved, where the 
30 viewer actually "feels" the distance and angle that he or 
she is with respect to the object. 

In the third embodiment, the synthetic image, as 
viewed by human eyes, is made to overlay the image of the 
real object over the useful three-dimensional space, and 
35 not merely over a single focal plane, as in the second 
embodiment. 

The means for generating synthetic stereoscopic pairs 
of pictures can be performed via liquid crystal displays 



25 
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(LCDs) , or other similar ways. Note that cathode ray 
tubes (CRTs) generally do not provide the amount of light 
and resolution necessary to superimpose a synthetic image 
onto a real, bright image, and so that is why LCDs are 
5 preferred in each of the embodiments described herein. 
In each of the embodiments, fiber optics are used in 
order to provide a light conduit between the light source 
and the liquid crystal displays, thereby allowing a 
strong source of light to be placed at a distance. The 
10 present invention may also include imaging means for 
imaging the distal tip of the optical fiber at the 
entrance pupil of the viewing ocular system, to allow an 
efficient coupling of light to the eye. The present 
invention may include adjusting means for adjusting the 
15 LCD illumination intensity, so as to equalize the left 
and right LCD brightness level. The light adjusting 
means may be implemented within the light sources that 
provide light for the LCDs. By proper light adjusting, 
a common brightness level can be made compatible with the 
20 illuminated field of view of the microscope. 

In the second embodiment, the left and right 
synthetic images are paraf ocalized to a common reference 
plane, while in the third embodiment, the left and right 
synthetic images are paraf ocalized to an optimum 
25 reference plane in the real object space. The 
magnification of the left and right synthetic images can 
be adjusted to match each other, via an adjusting means 
for left and for the right images. The magnification of 
the LCD display can also be adjusted via the optics 
30 provided in the LCD and Optics units, in order to adjust 
the ocular field of view to be covered by the display 
images . 

The present invention allows for calculating 
stereoscopic views from a data base defining: i) the 
35 synthetic three dimensional object, and ii) pre- 
calibrated correspondence between the three-dimensional 
space and the points in the plane of the LCDs. The data 
base is preferably stored in a memory that is accessible 
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by a computer. The computer preferably has embedded 
software for performing the above-mentioned calculation 
of the stereoscopic views. A transformation matrix may 
be utilized in order to determine which pixel in an X-Y 
5 array of pixels would be illuminated based on a 
particular angle and distance of the viewing system with 
respect to the real object. 

The invention allows for the ability to adjust the 
contrast of the pictures produced by the LCDs (i.e., 
10 either lightening them or darkening them, as needed) , 
based on the brightness of the real image from the real 
object. 

The invention also may be configured with X, Y and 
theta adjusting means for making X, Y and theta 
15 adjustments so as to allow for an adjustable overlay of 
the left and right eye views of the synthetic images 
within the field of view of the ocular (i.e., zoom in or 
zoom out the size of the synthetic image overlayed on the 
real image) . 

20 The invention also may include measuring means for 

measuring the parameters of the system needed to overlay 
the left and right images. 

Also, the parameters of the optical system containing 
variable lenses may be dynamically measured in order to 

25 dynamically maintain the overlay of the synthetic images 
in the field of view. 

Still further, the present invention may also include 
dynamic measuring means for dynamically measuring the 
parameters of the optical system to dynamically maintain 

30 the overlay of the synthetic images in the field of view. 

While embodiments have been described herein, 
modification of the described embodiments may become 
apparent to those of ordinary skill in the art, following 
the teachings of the invention, without departing from 

35 the scope of the invention. For example, the present 
invention has many applications, including: image guided 
surgery, image guided therapy, imaged guided diagnostics, 
telesurgery, robotic surgery, and inspection system to 
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compare real objects with their three dimensional 
reference profiles. 
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What Is Claimed Is : 

1. A method of stereoscopic viewing of a real 
object and a synthetic object, the method comprising the 
steps of : 

a) illuminating the real object; 

b) receiving light reflected off of the real 
object; 

c) creating a left eye and right eye stereoscopic 
synthetic image; 

d) combining the left eye stereoscopic synthetic 
image with the received light from the real object as a 
first combined image; 

e) combining the right eye stereoscopic synthetic 
image object with the received light from the real object 
as a second combined image; and 

f) providing the first combined image to a first 
eye of a viewer and providing the second combined image 
to a second eye of the viewer. 

2. The method according to claim 1, wherein the 
left eye and right eye stereoscopic synthetic images are 
created based on a computer output of video data, and 

wherein the video data is converted to the left and 
right eye stereoscopic synthetic image by providing the 
video data to a first and a second liquid crystal display 
(LCD) , respectively. 

3. The method according to claim 2, wherein a light 
source provides light for the first and second LCDs, 
thereby providing a particular intensity for the left eye 
and right eye stereoscopic synthetic images. 

4. The method according to claim 3, wherein the 
particular intensity is adjustable. 

5. The method according to claim 2, wherein the 
left eye and right eye stereoscopic synthetic images are 
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mapped onto a two dimensional space corresponding to a 
particular pixel of a matrix of pixels of the first and 
second LCDs, respectively, 

6- The method according to claim 5, wherein the 
particular pixel of the matrix of pixels of the first and 
second LCDs is actuated to create the left eye and right 
eye stereoscopic synthetic images, irrespective of an 
actual angle of the real object with respect to the 
viewer. 

7. The method according to claim 5, wherein the 
particular pixel of the matrix of pixels of the first and 
second LCDs is actuated to create the left eye and right 
eye stereoscopic synthetic images, based in part on an 
actual angle of the real object with respect to the 
viewer . 

8- An apparatus for stereoscopic viewing of a real 
object and a synthetic object, comprising: 

an illuminator for illuminating a real object; 

first and second objective lenses for receiving light 
reflected off of the real object; 

a computer for creating a left eye and right eye 
stereoscopic synthetic image and for respectively 
outputting a first and second video signal as a result 
thereof ; 

a first and a second liquid crystal display (LCD) for 
respectively receiving the first and second video signals 
and for providing a first and second synthetic image; 

a first combiner for combining the first synthetic 
image with the received light from the first objective 
lens as a first combined signal; 

a second combiner for combining the second synthetic 
image with the received light from the second objective 
lens as a second combined signal; and 

first and second oculars for respectively receiving 
the first and second combined signals and for 
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respectively directing the first and second combined 
signals to a left eye and a right eye of a viewer/ so as 
to achieve a stereoscopic viewing, 

9. The apparatus according to claim 8, wherein the 
first and second LCDs each comprise a two-dimensional 
array of pixels, 

10. The apparatus according to claim 8, further 
comprising: 

a light source for outputting light; and 

at least one fiber optic line for providing the light 

output from the light source to the first and second 

LCDs, 

wherein the fiber optic line provides the light to 
the LCDs for creation of the left eye and right eye 
stereoscopic images . 

11. " The apparatus according to claim 10, further 
comprising light adjusting means for varying an intensity 
of the light output from the light source, so as to vary 
an intensity of the left eye and right eye stereoscopic 
synthetic images. 

12. The apparatus according to claim 8, wherein the 
left eye and right eye stereoscopic synthetic images are 
mapped onto a two dimensional space corresponding to a 
particular pixel of a matrix of pixels of the first and 
second LCDs, respectively, 

13. The apparatus according to claim 12, wherein the 
particular pixel of the matrix of pixels of the first and 
second LCDs is actuated to create the left eye and right 
eye stereoscopic synthetic images, irrespective of an 
actual angle of the real object with respect to the 
viewer. 
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14. The apparatus according to claim 12, wherein the 
particular pixel of the matrix of pixels of the first and 
second LCDs is actuated to create the left eye and right 
eye stereoscopic synthetic images, based in part on an 
actual angle of the real object with respect to the 
viewer . 
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